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LONG  TERM  GOALS 

To  see  surface  ships  or  obstacles,  a  submarine  must  rise  close  to  the  surface  and  expose  a  periscope. 
This  sacrifices  the  submarine’s  inherent  stealth,  limits  its  submerged  speed,  and  raises  the  risk  of 
collision.  The  long-term  goal  of  the  Virtual  Periscope  investigation  is  to  develop  the  technology  that 
enables  submarines  to  detect  surface  ships  and  obstacles  from  tactically  useful  depths  and  ranges. 


Figure  1:  Virtual  Periscope  concept  of  operations  (left)  and  signal  processing  overview  (right). 


OBJECTIVES 

The  scientific  objectives  of  this  investigation  were  far-reaching  and  included  (a)  Advance  the  through- 
surface  viewing  algorithm  to  demonstrate  detection  in  simulated  data  of  targets  subtending  between  2 
and  3  degrees  in  elevation;  (b)  Make  the  algorithm  ready  for  at-sea  demonstration  in  the  form  of  post¬ 
processing  underwater  imagery  and  meta  data,  performing  the  necessary  motion  compensation,  wave 
estimation,  and  likelihood  detection  for  useful  through-surface  performance;  (c)  Design,  build,  and 
pool-test  the  data  acquisition  system  to  be  used  on  a  seagoing  platform;  (d)  Install  the  data  acquisition 
system  on  a  submarine,  conduct  data  collection  at  sea,  and  perform  post-processing  to  demonstrate 
useful  through-surface  imaging. 
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APPROACH 


The  Virtual  Periscope  TEMP  ALT  installation  consists  of  a  single  upward-looking  optical  sensor 
mounted  atop  the  sail,  a  portable  control  and  data  acquisition  system  used  in  the  Control  Room  during 
data  collection,  a  hybrid  fiber-optic  electrical  hull  penetrator  (installed  in  place  of  the  AN/BLD- 1 
electrical  hull  fitting),  and  interconnecting  cables.  An  overall  diagram  is  provided  in  Figure  2. 
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Figure  2:  Virtual  Periscope  data  collection  system  as  installed  on  USS  CHICAGO  (SSN-721) 


The  sensor  housing  is  mounted  in  the  lookout  station  atop  the  sail.  The  station’s  existing  hatch  cover  is 
restrained  in  the  open  position,  with  the  sensor  foundation  providing  a  hydrodynamic  fairing  for  the 
sail.  The  acrylic  dome  port’s  hemispherical  shape  protrudes  three  inches  above  the  sail  surface.  When 
not  in  use  the  dome  can  be  covered  and  protected  from  scratches  by  bolting  on  the  stainless  steel  cover. 
The  purpose  of  the  dome  port  is  to  remove  the  magnification  that  occurs  at  the  air-water  interface  of  a 
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flat  port,  and  allow  the  camera  to  capture  the  full  140-degree  field  of  view  needed  to  capture  all  the 
refracted  light  penetrating  the  ocean  surface  and  reaching  the  sensor. 

Within  the  sensor  housing  is  a  Dalsa  1M30  CCD  camera,  fisheye  lens,  and  interface  electronics  that 
collect  underwater  images  and  transmit  them  to  the  data  acquisition  computer  in  the  Control  Room  for 
storage  and  post-processing.  The  FPGA  circuitry  inside  the  housing  performs  framegrabbing,  fiber 
optic  conversion,  and  remote  control  functions.  The  lens  has  a  remotely  controlled  aperture  to  allow 
for  collection  in  variable  lighting  conditions.  The  underwater  cable  connected  to  the  housing  provides 
24  VDC  power  via  copper  conductors,  and  fiber  optic  transmission  for  data  output  and  control 
functions. 

The  Portable  Data  Acquisition  Computer  (PD AC)  is  a  self-contained  workstation  with  integrated 
keyboard  and  monitor.  With  it  the  operator  performs  the  data  acquisition  and  control  functions  needed 
for  post-processing  and  demonstration  of  through-surface  viewing.  The  PD  AC  is  connected  to  the 
electrical  hull  fitting  via  stowable  umbilical,  and  to  the  RLGN  via  stowable  Ethernet  connection. 
Located  atop  the  RLGN  CDU  chassis,  the  unused  J1 3  connection  streams  binary  navigation  data 
needed  for  image  motion  compensation  to  the  PD  AC.  The  PD  AC  stores  image  and  navigation  data  to 
removable  hard  drives  housed  in  the  PD  AC  chassis.  Each  hard  drive  holds  about  two  hours  of  data, 
sufficient  for  multiple  data  collection  runs  before  shutting  down  the  computer  and  swapping  disks. 

The  algorithm  approach  to  the  Virtual  Periscope  is  as  outlined  in  Figure  3.  The  data  acquisition  system 
collects  underwater  images  of  the  entire  Snell  cone  of  light  that  penetrates  the  ocean  surface.  At  a  rate 
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Figure  3.  The  algorithmic  approach  to  through-surface  visualization  and  detection. 


3 


of  10  frames  per  second,  the  camera  sends  digital  images  to  the  onboard  computer  for  registration  to  an 
earth  reference  using  the  navigation  information  provided  by  the  ship’s  inertial  navigation  system.  The 
radial  component  of  the  ocean  slope  field  is  measured  in  each  of  the  registered  images  by  determining 
the  position  of  extinction  boundaries  between  groups  of  transmitting  and  internally  reflecting  pixels. 
The  sparse  set  of  radial  slope  measurements  is  then  filtered  in  space  and  time  to  provide  both  radial 
and  transverse  slope  estimates  over  the  entire  measurement  region  through  the  process  of  wave 
estimation.  With  the  slope  estimates  thus  obtained,  the  images  are  processed  through  either  or  both  of 
visualization  and  detection  streams.  For  visualization,  the  images  are  raytraced  using  the  slope 
estimates  to  reduce  the  distortions  due  to  the  resolvable  (wavelength  longer  than  a  pixel)  surface  waves 
and  subsequently  deconvolved  to  reduce  the  blur  due  to  the  unresolved  (subpixel)  surface  waves.  For 
detection,  the  images  and  the  estimated  slope  fields  are  input  to  a  statistical  calculation  that  computes 
the  likelihood  that  a  ship  silhouette  is  occluding  light.  The  output  of  the  detector  can  be  presented  in 
the  form  of  a  waterfall  display  that  presents  the  likelihood  as  a  function  of  time  and  bearing. 

WORK  COMPLETED 

Upon  completing  the  underwater  data  collection  system,  the  team  conducted  an  initial  pool  test  to 
check  system  watertight  integrity,  conduct  submerged  testing  of  the  data  acquisition  system  in  various 
lighting  and  sky  conditions,  and  check  system  focus  in  the  underwater  environment.  As  shown  in 
figure  4,  the  pool  test  vividly  demonstrates  that  in  still  water,  the  through- surface  view  can  be  readily 
remapped  to  Cartesian  coordinates  and  the  entire  above-surface  reproduced  with  excellent  clarity. 

Once  the  water  surface  is  ruffled  (in  this  case  with  a  leaf  blower),  fidelity  of  the  above-surface  view  is 
rapidly  degraded.  All  objectives  of  the  pool  test  were  fully  met,  and  the  team  was  able  to  submarine 
installation  and  ocean  testing. 


Figure  4.  Initial  pool  test  of  the  Virtual  Periscope  data  collection  system,  showing 
the  sensor  setup  (left),  the  remapped  and  undisturbed  surface  view  (top  and  center), 
and  the  remapped  and  wind-blown  surface  view  (right  and  bottom). 
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The  full  complement  of  Virtual  Periscope  testing  consisted  of  a  seven  day  transit  from  San  Diego  to 
Pearl  Harbor,  followed  by  a  single  day  of  testing  in  the  waters  south  of  Oahu  with  a  dedicated  surface 
target.  The  test  schedule  was  comprised  of  several  runs  of  one  of  four  prescribed  geometries:  parallel, 
circular,  racetrack  (250-yard  CPA),  and  racetrack  (500-yard  CPA).  The  surface  vessel  providing  target 
services  was  USS  LASSEN  (DDG-82).  LASSEN  was  generally  tasked  to  be  dead-in-the -water  (DIW) 
or  (in  the  case  of  the  parallel  geometries)  to  maintain  a  steady  course  and  speed.  USS  CHICAGO  ran 
the  assigned  geometry,  observing  the  target  with  the  submerged  sensor  while  recording  imagery  and 
navigation  data  with  the  TEMP  ALT  acquisition  system. 


The  Parallel  Geometry  shown  in  Figure  5  required  the  surface  vessel  to  maintain  a  steady  course  and 
speed  of  2,  4,  or  6  knots.  The  submarine,  operating  at  a  keel  depth  of  120  or  150  feet,  operated  abeam 
on  the  same  course  and  speed  for  several  five-minute  collection  periods  at  various  ranges.  The 
submarine  used  a  combination  of  active  and  passive  sonar  for  station  keeping. 
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Figure  5:  The  Parallel  Geometry  for  Virtual  Periscope  ocean  testing. 

The  Circular  Geometry  shown  in  Figure  6  required  that  the  surface  vessel  remain  dead  in  the  water 
(DIW).  The  submarine,  operating  at  a  keel  depth  of  120  or  150  feet,  circled  the  target  three  times. 
Depth  separation  between  the  submarine  and  surface  ship  is  maintained  throughout. 
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Figure  6:  The  Circular  Geometry  for  Virtual  Periscope  ocean  testing. 


The  team  collected  eight  hard  disks  (about  one  terabyte)  of  excellent  image,  navigation,  and  fire 
control  data.  CHICAGO  and  LASSEN  supported  the  test  fully,  and  we  completed  the  planned  parallel, 
racetrack,  and  circular  geometries  at  ranges  from  0  to  4000  yards.  Several  runs  brought  CHICAGO  to 
pass  directly  beneath  with  spectacular  results.  The  night  data  was  done  under  a  96%  moon  and  clear 
skies  at  150  feet,  PD,  and  broaching.  Wave  contrast  was  very  good  by  day,  and  through-surface  moon 
visibility  leads  us  to  believe  that  our  algorithm  will  successfully  detect  navigation  lights  at  night. 
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Figure  7:  Summary  of  data  collected  in  at-sea  operations  of  the  Virtual  Periscope.  Clockwise  from 
upper  left  is  broaching  data;  imagery  from  sensor  depth  of  10  feet;  imagery  from  sensor  depth  of 
100  feet;  data  collected  at  night;  daytime  data  collected  from  600 feet  sensor  depth;  and  data 
collected  from  100  feet  sensor  depth  with  target  hull  coming  into  view. 


In  Figure  7  above,  the  breadth  and  extent  of  data  collected  at  sea  for  post  processing  analysis  can  be 
seen.  Perhaps  most  surprisingly,  the  sun  can  still  be  clearly  seen  from  a  sensor  depth  of  600  feet.  While 
not  much  can  be  seen  of  individual  wave  facets,  the  sun’s  rays  do  in  fact  penetrate  to  great  depths  and 
future  investigations  may  find  ways  to  exploit  this  information.  Also  of  immediate  interest  is  the  much 
different  distortion  effects  taking  place  when  the  sensor  is  just  beneath  the  surface  at  broaching  depth. 
This  may  lead  to  unique  algorithms  that  take  advantage  of  the  large  tilting  wave  face  and  imaging 
above  the  region  of  first  caustic  focus.  Contrasting  the  10  foot  versus  the  100  foot  deep  imagery 
provides  great  insight  into  the  scattering  effects  in  the  water  column  that  must  be  accounted  for  in  the 
wave  estimation  and  detection  processes. 

RESULTS 

The  data  selected  for  our  initial  demonstration  takes  place  near  sunset,  with  CHICAGO  up-sun  and 
slowly  closing  LASSEN  until  passing  directly  underneath.  This  choice  was  driven  by  the  facts  that 
these  images  had  a  manageable  level  of  scattering  of  solar  illumination  (and  hence  good  wave 
visibility)  and  that  they  were  in  the  vicinity  of  a  LASSEN  passover,  enabling  an  accurate 
reconstruction  of  the  relative  position  of  the  ship  without  access  to  fire  control  data. 
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Except  for  a  small  set  of  analyses  using  San  Clemente  Island  data  collected  in  2001,  the  Virtual 
Periscope  algorithms  had  been  developed  using  simulated  data.  Transitioning  the  algorithms  for 
application  to  sea  test  data  presented  several  challenges.  First,  the  imagery  had  to  be  temporally 
synchronized  with  navigation  data  supplied  by  the  RLGN  system  so  that  the  imagery  could  be  properly 
registered  to  an  absolute  spatial  and  angular  coordinate  system.  Using  the  RLGN  navigation  data  for 
platform  motion  compensation,  we  had  to  apply  corrections  for  lag  and  drift  between  the  imagery  and 
the  navigation.  This  was  accomplished  with  data  from  a  synchronization  maneuver  in  which 
CHICAGO  executed  a  10  degree  downward  pitch  followed  by  a  10  degree  upward  pitch.  By  tracking 
the  position  of  the  sun  during  this  maneuver  and  correlating  against  the  RLGN  data,  we  were  able  to 
determine  the  time  lag  to  within  a  few  tenths  of  a  second.  Taking  this  lag  into  account,  the  sun 
position  remained  fixed  in  the  registered  imagery  as  expected. 

A  second  challenge  in  the  sea  test  data  was  to  cope  with  the  highly  nonuniform  illumination  of  the 
scene.  Previously,  the  algorithms  had  been  designed  for  the  situation  of  a  completely  uniform  sky,  as 
might  be  encountered  on  an  overcast  day.  A  uniform  sky  simplifies  the  situation  considerably  in  a 
number  of  respects,  permitting  an  analytic  formulation  of  the  gradiometer  as  one  example.  With  the 
sun  on  the  horizon  and  clouds  in  the  sky,  the  selected  dataset  did  not  at  all  resemble  uniform 
illumination.  In  response  to  this  challenge,  we  developed  a  new  approach  to  gradiometry  (the  first- 
pass  determination  of  wave  slope  from  measured  brightness)  that  was  demonstrated  to  be  much  more 
robust  against  highly  variable  illumination  associated  with  sun  and  clouds.  Testing  this  approach 
against  simulations  with  non-uniform  skies,  we  found  much  better  performance  than  our  earlier 
gradiometry  technique  (designed  for  uniform  skies),  without  significant  performance  loss  when  the  sky 
is  actually  uniform.  The  technique  hinges  on  collecting  brightness  statistics  locally  within  an  image 
and  intelligently  interpolating  between  distributions  measured  in  different  locations.  We  applied  the 
new  gradiometer  to  the  LASSEN  data  and  generated  plausible  slope  measurements  across  the  entire 
Snell  cone.  A  similar  technique  was  also  built  into  our  likelihood  ratio  detector  to  improve 
performance  against  nonuniform  skies. 

A  third  challenge  was  to  compensate  for  the  motion  of  the  platform  in  the  wave  estimator.  Earlier 
work  with  simulations  and  with  the  San  Clemente  Island  dataset  had  been  based  upon  a  stationary 
sensor  at  a  fixed  depth.  We  developed  algorithms  for  motion  compensation  that  were  demonstrated  in 
simulations  with  a  moving  sensor  to  achieve  wave  estimation  accuracies  comparable  to  a  stationary 
camera  (second  order  effects  come  into  play  with  a  moving  sensor  that  depend  on  the  relative  speed 
and  direction  of  the  camera  with  respect  to  the  wave  field).  Using  the  RLGN  data  to  register  the 
imagery  and  applying  the  motion  compensation  algorithms,  we  produced  plausible  wave  estimates 
over  a  span  of  data  in  which  the  platform  is  performing  a  tight  circular  orbit. 

A  fourth  challenge  was  presented  by  nonlinear  wave  propagation.  This  is  a  complex  interaction  among 
waves  that  is  not  straightforward  to  include  within  the  wave  estimator.  We  have  been  concerned  about 
the  adverse  impact  of  nonlinear  waves  on  estimator  performance,  anticipating  that  it  would  be 
comparable  to  that  of  relief  distortion.  We  updated  our  simulation  with  a  new  capability  to  model 
nonlinear  surface  effects  using  a  simple  approximation  that  accurately  represents  nonlinearities  to  first 
order.  The  most  noticeable  effect  in  this  approximation  is  the  advection  of  short  waves  by  longer 
waves  and  the  sharpening  of  wave  crests.  We  found  that,  as  anticipated,  wave  estimator  performance 
was  significantly  worsened  when  this  effect  was  included  in  the  simulation.  We  addressed  this 
problem  by  building  compensation  routines  into  our  wave  estimator  and  have  found  that  performance 
can  be  partially  recovered  in  comparison  with  simulations  without  wave  advection.  With  further  work, 
a  complete  recovery  may  be  possible. 
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The  likelihood  detector  also  underwent  significant  revisions  in  the  course  of  analyzing  the  sea  test  data. 
In  a  similar  fashion  as  the  gradiometer  algorithm,  the  detector  was  also  generalized  to  cope  with  highly 
nonuniform  illumination.  It  was  also  generalized  to  detect  larger  targets  (i.e.  those  with  significant 
signal  inside  the  Snell  cone)  so  that  it  would  behave  well  with  the  LASSEN  at  close  range. 

After  converting  the  detector  algorithm  into  a  much  faster  C++  code,  we  processed  target-absent  and 
target-present  data  to  provide  the  likelihood  detection  algorithm  with  the  slope  and  brightness  statistics 
that  form  the  likelihood  ratio.  In  our  first  end-to-end  test,  we  applied  the  likelihood  detector  to  the 
wave  estimator  output  and  gained  a  tentative  though-surface  detection  on  LASSEN  well  before  the  hull 
emerged  in  the  raw  imagery.  The  detector  behaved  well,  detecting  the  LASSEN  at  a  range  of  130 
yards  in  spite  of  the  much  larger  signal  produced  by  the  sun  on  the  horizon. 


Instantaneous  Likelihood  Raw  Imagery 


Figure  8.  Initial  results  of  applying  the  detection  algorithm  to  the  ocean  target  data 
in  an  end-to-end  test.  To  the  left  is  the  instantaneous  likelihood  output,  displayed  as 
a  single  line  in  a  20-second  waterfall  display  at  top.  In  the  raw  imagery  the  sun  is 
most  visible,  with  the  reflected  glow  of  the  superstructure  coming  into  view. 

A  number  of  promising  areas  of  investigation  remain  for  us  to  extend  the  detection  limit  to  a  more 
tactically  useful  range.  One  of  the  most  prominent  is  the  effect  of  volume  scattering,  especially  when 
sunbeams  are  present.  While  we  have  successfully  compensated  for  sky  illumination  that  is  spatially 
nonuniform,  we  have  observed  that  temporal  fluctuations  in  the  illumination  as  direct  solar 
illumination  is  refracted  by  long  wavelength  surface  waves  produce  strong  responses  in  the 
gradiometer  and  likelihood  detector  that  are  unrelated  to  the  presence  of  the  target.  Both  sets  of 
algorithms  could  be  designed  to  cope  with  brightness  fluctuations  of  those  spatial  and  temporal  scales, 
leading  to  higher  quality  wave  estimates  and  more  sensitive  detector  output. 
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TRANSITIONS 


While  this  research  has  not  yet  transitioned  to  an  acquisition  or  operational  command,  it  is  being 
conducted  in  concert  with  the  submarine  imaging  acquisition  office  of  NavSea,  PMS  435.  It  is  also 
being  conducted  with  the  mutual  benefit  of  Low  Observable  Periscope  research  which  is  sponsored  by 
ONR,  DARPA,  and  NSWC  Panama  City. 


RELATED  PROJECTS 

Work  on  the  Virtual  Periscope  project  is  related  to  the  developmental  work  on  the  Low  Observable 
Periscope.  While  both  are  through-surface  sensors,  Virtual  Periscope  is  designed  for  a  sensor  depth  of 
150  feet  while  Low  Observable  Periscope  is  intended  for  a  depth  of  only  a  few  inches.  The  Virtual 
Periscope  effort  is  focused  on  creating  an  algorithm  that  can  detect  the  presence  of  a  surface  ships  at 
maximum  range  for  collision  avoidance.  By  operating  on  imagery  data  collected  from  only  a  few 
inches  deep,  the  Low  Observable  Periscope  algorithm  aims  to  produce  imagery  of  sufficient  clarity  to 
visualize  the  above-ocean  scene  for  covert  surveillance  purposes. 
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